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Longitudinal Data Systems to Support Data-Informed Decision Making: 
A Tri-State Partnership Between Michigan, Minnesota, and Wisconsin 

Christopher A. Thorn and Robert H. Meyer 



Many states are ill equipped to manage and effectively use the data that they are now 
required to collect as a result of the No Child Left Behind Act (NCLB; 2002) and other state and 
federal policy initiatives. Michigan, Minnesota, and Wisconsin, in partnership with the 
Wisconsin Center for Education Research (WCER), recently submitted a collaborative proposal 
(Wilmot & Meyer, 2005) to the U.S. Department of Education, Institute of Education Sciences, 
to create a multistate longitudinal data system (EDS) that will enable education stakeholders to 
conduct value-added and other diagnostic and policy-relevant evaluation research and engage in 
data-informed decision making, with the ultimate goal of strengthening teaching and improving 
student achievement for all students and all schools. All three states received awards, creating a 
joint initiative called the Tri-State Partnership. 

To achieve its goals, the Tri-State Partnership is structured to ensure that all EDS design 
decisions will be fully informed by a thorough understanding of end-use requirements and, more 
generally, by the needs of all education stakeholders: parents and students; teachers; school, 
district, and state leaders and program staff; and policy makers (Hamalainen, 2001). By reaching 
out to a wide range of stakeholders, we expect to foster a sense of shared purpose and ownership 
in the EDS, resulting in stakeholder buy-in to the strategy of using data to drive student 
achievement. 



Leveraging a Partnership to Do System Design 

One of the distinctive aspects of the Tri-State Partnership is that it reflects genuine 
collaboration among three states and WCER. Working together will permit each state to share 
responsibility for at least 50% of all project tasks — in effect more than doubling the impact of the 
resources allocated. Moreover, by structuring work products so that they can be shared across the 
partnership, we expect to create products that will be of value to states outside the partnership 
(Alter, 2002). Project results and products, including overview papers that describe the concepts 
and strategies used in the project, will also be disseminated via conferences and workshops, 
including the Management Information Systems conferences sponsored by the U.S. Department 
of Education, National Center for Education Statistics. 

The EDS developed by the partnership will include data on all students enrolled in PK- 
12 education. Collaborating with state partners in the PK-16-i- Initiative, we expect to extend the 
EDS to include students enrolled in higher education. ^ This work will be a critical component of 
efforts focused on identifying the relative of success of high school graduates in the transition to 
postsecondary education. This partnership will also explore the data and collaborative 
infrastructure needed to evaluate teacher education programs. This would include linking to the 



* The PK-16 Initiative was created to support collaboration between PK-12 and postsecondary education 
institutions. The goals of the group include improving the quality of PK-12 teacher preparation and professional 
development and aligning high academic standards for PK-12 students with postsecondary education. 
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Student outcome data from new teachers in the PK-12 system in order gauge the effectiveness of 
teacher preparation programs. 

The project is divided into distinct task areas — (a) data analysis and research 
requirements; (b) data access; (c) data dictionary; (d) data warehouse; and (e) secure data 
transport — as illustrated in Figure 1 . Specific research methods for each task area are presented 
in detail in the proposal (Wilmot & Meyer, 2005). Each of the partners has been assigned a 
specific level of responsibility for each task and subtask: (a) primary responsibility; (b) 
secondary responsibility; or (c) review and implementation. The partners are also committed to 
using computer-supported collaborative work (CSCW) tools — that is, collaborative tools that 
will support distributed work and address many of the challenges of multisite development 
(Grudin, 1990, 1994). CSCW tools were used in the writing of the proposal as well as in the 
development of management plans. This early use of CSCW tools by both leadership and project 
team members created a shared understanding of the work practices involved. 




Figure 1. Longitudinal data system mind map (Wilmot & Meyer, 2005, p. 7). 
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Laying out the Details of Development 

Tasks and subtasks of the partnership are broken down into discrete parts that can be 
designed and implemented in phases. This incremental approach to implementation will help us 
launch end-use applications and build local support for the overall project more quickly and 
effectively. Three types of activities will be subject to this phased-in implementation: 

1. Shared cross-state activities (such as development and adoption of data dictionary 
specifications); 

2. State-specific implementation of products (such as the data portal to support data access by 
various stakeholders); and 

3. End-use applications (such as piloting a state value-added system). 

Figure 1 is a concrete example of the initial development work done as part of the 
proposal preparation process and the collaborative process the partnership will engender. The 
graphical EDS “mind map” in Figure 1 was a contribution of one of the partners (Minnesota) to 
the others. Minnesota had already done substantial work with an external contractor on the major 
dimensions of an EDS. They generously allowed staff from the other states to take advantage of 
this investment and leverage it for the development of lines of work for the partnership. The 
collaborative technologies supported by WCER made this sharing easy and secure (Thom, 

2005). The availability of collaborative solutions coupled with the human support needed to 
scaffold their adoption allowed the partnership to engage in extensive knowledge accumulation 
and sharing. 



Adopting Complex Tools and Evaluation Models 

In developing longitudinal data systems to support data-informed decision making, the 
Tri-State Partnership is taking on a number of challenging tasks, made all the more difficult by 
the need for effective communication and collaboration among the partner states. This burden 
will have at least two dimensions. One will be to identify the decision support needs of the many 
stakeholder groups and to align storage and representation models with those needs. The other 
will be to foster understanding of new value-added models of program evaluation. Each of these 
challenges is discussed below. 

Payoffs to Developing New Storage and Representation Models 

The achievement of the partnership goals will require increasing understanding by the 
partners of the return on investment in knowledge infrastmcture. One key payoff of such 
investment is the ability to develop new storage and representation models designed to support 
the decision support needs of partners’ many stakeholder groups (Ross & Kimball, 2006). 

The difficulty of developing new ways of representing data to inform decision making 
should not be underestimated. Muthukumar and Hedberg (2005) identify two important products 
of data warehouse design and construction: (a) the knowledge repository and (b) the knowledge 
refinery. The repository is “an electronic model that stores and manages explicit knowledge 
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resources” (p. 384). The refinery is “a representation of the activities involved in the electronic 
creation and distribution of the explicit digital knowledge resources contained in the repository” 
(p. 384). These are important distinctions that are not transparent to someone who has not 
previously grappled with longitudinal data warehouse design. 

The repository is a departure from traditional models in which data management is 
organized around operations. Early state-level education systems were designed to manage and 
track licensing for adult professionals (teachers, counselors, superintendents, etc.). Other systems 
were built around particular funding streams and were designed to ensure that services delivered 
to designated populations — special education students, English language learners, vocational 
education students, or other student groups — complied with funding agency requirements. Early 
state-level testing systems were designed to collect the results of statewide testing. Typically, 
testing was done in a handful of grades, and results were reported to the federal government for 
compliance purposes. Over the past 5-10 years, many states have also used test data to publish 
(in paper and on the Web) the year-by-year percentage of students’ achieving proficient or 
average test scores to inform the public about the average performance of schools and districts 
by subgroup (gender, race, ethnicity, language status, etc.). 

It is in the area of student outcome data — test results as well as behavioral indicators such 
as attendance and discipline statistics — that the first repositories began to emerge. Even then, 
most states did not link student performance over time in order to show growth in individual 
student learning. A repository, in contrast, is designed to link data from different operational 
sources into relational models that allow new meaning to be made of the data. 

A refinery may transform data for a new audience or impose operational rules that differ 
from those defined for reporting systems. The following examples illustrate two such differences 
and the way in which they might be used to support decision making in particular circumstances. 

Eirst, consider the often-used proxy for household income — eligibility for free and 
reduced-price lunch. Some evidence suggests that students experience social stigma when they 
participate in free lunch (and breakfast) programs (California Project Eean, 2005). This 
phenomenon is associated with age: the older children are, the more likely they are to perceive 
this stigma. If children do not avail themselves of free or reduced-price meals in order to avoid 
the associated stigma, it follows that free and reduced-price lunch eligibility codes would 
underrepresent the number of poor students. Eurther, it is likely that this underreporting would 
increase with age. In the decision support system (aka refinery), it might be more appropriate to 
look at historical data to see if a child has ever been coded as receiving free or reduced-price 
lunch. This approach might yield a better indicator of household income and address both the 
issue of underreporting and concerns about the long-lasting impact of poverty. What would be a 
misrepresentation for an operational data system associated with food services would be totally 
appropriate for research on program effectiveness in which it was important to control for 
household income. 

A second example comes from another area of household data. During the annual 
registration process, some districts and states request — but do not require — data on parental 
education. Because the data is not required, parents often decline to provide it. Given that most 
student systems are able to handle basic geographical data — zip plus four or census tract and 
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block — it would be possible to use average edueation data from census reeords to fill in missing 
values. The appropriateness of using sueh a substitution technique depends on its purpose. If 
large consequences are associated with a decision made with this data, concerns about 
misrepresentation would be large. However, if the eonsequenees are small, the inelusion of 
household edueation might provide vital information for redrawing attendanee boundaries or 
other grouping activity intended to address some equity or access issue. 

For state agencies whose primary information technology activity has focused on 
eompliance, shifting to a model that allows for exploration and evaluation will not be a simple 
task. However, the payoffs of this increased flexibility are likely to be quite large. 

Payoffs to Adopting New Value-Added Models of Program Evaluation 

One of the challenges of the Tri-State Partnership — aside from the daunting task of 
building a complex student data system able to link students to teachers, schools, and courses — 
will be to encourage the partner states to explore new models of program evaluation. The Value- 
Added Research Center (VARC) at WCER is the project home for this effort. The VARC 
leadership team (the authors of this paper) will work with the three state partners to foster new 
ways of thinking about student outcomes and program effectiveness (Meyer, 2000; Meyer, 
2002 ). 
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Figure 2. General value-added model for a given subjeet, grade, and year (Wilmot & Meyer, 
2005, p. 113). 

Figure 2 represents what we refer to as a general value-added model. It does not specify 
the particular value-added teehnique used to address a speeific testing and aecountability regime. 
Rather, it is our attempt to make explicit our understanding of a robust model of growth in 
student learning. The notational system is intended for technical audiences, but the model can 
also be explained using the boxes below the figure equation. What we are arguing is that models 
of performanee should explieitly address the impaet of student eharaeteristics and the effeet of 
classroom, school, and district inputs. This differentiates the VARC model from other value- 
added analysis frameworks that relegate many of these factors to a black box or fail to address 
them at all. 
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The VARC approach to school and district data analysis does not abandon the NCLB 
mantra of high expectations. Rather, the VARC model supports high expectations by examining 
the value added by programs and materials, professional development and teacher training, and a 
variety of models for school and classroom organization to determine which children learn more 
under which settings. High expectations by themselves provide a motivator but not guidance on 
how best to achieve challenging learning goals. As explained in the partnership proposal (Meyer 
& Wilmot, 2005), VARC’s general value-added/longitudinal model can be used to estimate both 
value-added indicators and the effects of educational programs and other inputs. Thus, a single 
model can be used both to achieve educational accountability and to evaluate program 
effectiveness. One of the end-use applications that the partnership will explore is combining 
these tasks so that it is possible to provide concrete guidance to educators about available 
program options that will improve the performance of classrooms and schools. Each state in the 
Tri-State Partnership will examine different value-added models that are appropriate to its own 
testing schemes, resource allocation models, and governance structures. 

As with the introduction of a refinery model of system design, the introduction of large- 
scale evaluation to the partnership will pose a number of technical and social challenges. Gaps in 
conceptual knowledge of study design and the appropriate use of student outcome data to fairly 
evaluate programs, schools, and materials are likely to engender concern across the entire range 
of stakeholders. It will be the responsibility of the partnership members to anticipate these 
concerns and focus on the shared values of rigor, transparency, and organizational improvement 
that brought the teams together. 

The EDS proposal is an ambitious effort to address technical, social, and political 
challenges in supporting educational improvement. The Tri-State Partnership has been designed 
to address these challenges and create a collaborative environment that focuses on the shared 
needs and interests of the partners - maximizing the opportunity for collective action. The policy 
environment created by NCEB provides the impetus for change. It is our plan to exploit this 
pressure to race to the top rather than fight a rear guard action. 




Longitudinal Data Systems to Support Data-Informed Decision Making 



References 

Alter, S. (2002). Whiteboard: The eollaboration triangle. CIO Insight, 9, 21-27. Retrieved 
December 22, 2005, from http://www.cioinsight.eom/article2/0, 1540, 16484,00. asp 

California Project Lean. (2005). Student wellness: A healthy food and physical activity policy 
resource guide. West Sacramento: California School Boards Association. Retrieved 
December 21, 2005, from http://www.csba.org/ps/hf.htm 

Grudin, J. (1990). Groupware and cooperative work: Problems and prospects. In B. Laurel (Ed.), 
The art of human-computer interface design (pp. 171-185). Reading, MA: Addison- 
Wesley. 

Grudin, J. (1994). Groupware and social dynamics: Eight challenges for developers. 
Communications of the ACM, 57(1), 92-105. 

Hamalainen, R. (2001). Evaluating a framework for multi- stakeholder decision support in water 
resource management. Group Decision and Negotiation, 10(4), 331-353. 

Meyer, R. H. (2000). Value-added indicators: A powerful tool for evaluating science and 

mathematics programs and policies. NISE Brief, 3(3), 1-7. Retrieved December 22, 2005, 
from http://www.wcer.wisc.edu/archive/nise/Publications/Briefs/Vol 3 No 3/ 
Vol%203.%20No%203.pdf 

Meyer, R. H. (2003). Value-added indicators: Do they make an important difference? Evidence 
from the Milwaukee Public Schools (WCER Working Paper No. 2003-5). Madison: 
University of Wisconsin-Madison, Wisconsin Center for Education Research. Retrieved 
December 21, 2005, from http://www.wcer.wisc.edu/publications/workingPapers/ 
Working Paper No 2003 5.php 

Muthukumar, S. E., & Hedberg, J. G. (2005). A knowledge management technology architecture 
for educational research organisations: Scaffolding research projects and workflow 
processing. British Journal of Educational Technology, 36(3), 379-395. 

No Child Eeft Behind Act of 2001, Pub. E. No. 107-110, 1 15 Stat. 1425 (2002). 

Ross, M. & Kimball R. (2004, March 6). Differences of opinion: Comparing the dominant 

approaches to enterprise data warehousing. Intelligent Enterprise. Retrieved March 3, 
2006, from http://intelligententerprise.com/show Article. ihtml?articleID= 17800088 

Thorn, C. (2005). Systemic reform efforts in the U.S.: Role of information technology in 

fostering collaboration within new partnerships. In A. Visscher, J. Osorio, & Arthur 
Tatnall (Eds.,) Information technology in educational management. Dordrecht, 
Netherlands: Kluwer Academic Publishers. 

Wilmot, B., & Meyer, R. H. (2005). Longitudinal data systems to support data-driven decision- 
making: A proposal from Wisconsin (proposal submitted to the Institute of Education 
Sciences Statewide Eongitudinal Data System Grants Program). Madison: University of 



9 



Longitudinal Data Systems to Support Data-Informed Decision Making 



Wisconsin-Madison, Wisconsin Center for Education Research, Value-Added Research 
Center. Retrieved December 23, 2005, from http://www.wcer.wisc.edu/varc/pubs/ 

DPI WCER EDS Proposal.pdf 



10 



